Integrated Publishing Toolkit Research Articles

"Publish First": A Rapid, GPT-4 Based Digitisation System for Small Institutes with Minimal Resources

We present a streamlined technical solution ("Publish First") designed to assist smaller, resource-constrained herbaria in rapidly publishing their specimens to the Global Biodiversity Information Facility (GBIF). Specimen data from smaller herbaria, particularly those in biodiversity-rich regions of the world, provide a valuable and often unique contribution to the global pool of biodiversity knowledge (Marsico et al. 2020). However, these institutions often face challenges not applicable to larger herbaria, including a lack of staff with technical skills, limited staff hours for digitization work, inadequate financial resources for specialized scanning equipment, cameras, lights, and imaging stands, limited (or no) access to computers and collection management software, and unreliable internet connections. Data-scarce and biodiversity rich countries are also often linguistically diverse (Gorenflo et al. 2012), and staff may not have English skills, which means pre-existing online data publication resources and guides are of limited use. The "Publish First" method we are trialing, addresses several of these issues: it drastically simplifies the publication process so technical skills are not necessary; it minimizes administrative tasks saving time; it uses simple, cheap and easily available hardware; it does not require any specialized software; and the process is so simple that there is little to no need for any written instructions. "Publish first" requires staff to attach QR code labels containing identifiers to herbarium specimen sheets, scan these sheets using a document scanner costing around €300, then drag and drop these files to an S3 bucket (a cloud container that specialises in storing files). Subsequently, these images are automatically processed through an Optical Character Recognition (OCR) service to extract text, which is then passed on to OpenAI's Generative Pre-Transformer 4 (GPT-4) Application Programming Interface (API), for standardization. The standardized data is integrated into a Darwin Core Archive file that is automatically published through GBIF's Integrated Publishing Toolkit (IPT) (GBIF 2021). The most technically challenging aspect of this project has been the standardization of OCR data to Darwin Core using the GPT-4 API, particularly in crafting precise prompts to address the inherent inconsistency and lack of reliability in these Large Language Models (LLMs). Despite this, GPT-4 outperformed our manual scraping efforts. Our choice of GPT-4 as a model was a naive one: we implemented the workflow on some pre-digitized specimens from previously published Norwegian collections, compared the published data on GBIF with GPT-4's Darwin Core standardized output, and found the results satisfactory. Moving forward, we plan to undertake more rigorous additional research to compare the effectiveness and cost-efficiency of different LLMs as Darwin Core standardization engines. We are also particularly interested in exploring the new "function calling" feature added to the GPT-4 API, as it promises to allow us to retrieve standardized data in a more consistent and structured format. This workflow is currently under trial in Tajikistan, and may possibly be used in Uzbekistan, Armenia and Italy in the near future.

An Australian Model of Cooperative Data Publishing to OBIS and GBIF

The Australian Commonwealth Science and Industrial Research Organisation (CSIRO) hosts both the Australian Ocean Biodiversity Information System (OBIS) and Global Biodiversity Information Facility (GBIF) nodes within the National Collections and Marine Infrastructure (NCMI) business unit. OBIS-AU is led by the NCMI Information and Data Centre and publishes marine biodiversity data in the Darwin Core (DwC) standard via an Integrated Publishing Toolkit (IPT), with over 450 marine datasets at present. The Australian GBIF node is hosted by a separate team at the Atlas of Living Australia (ALA), a national-scale biodiversity analytical and knowledge delivery portal. The ALA aggregates and publishes over 800 terrestrial and marine datasets from a wide variety of research institutes, museums and collections, governments and citizen science agencies, including OBIS-AU. Many OBIS-AU published datasets are harvested and republished by ALA and vice-versa. OBIS-AU identifies, performs Quality Control and formats marine biodiversity and observation data, then publishes directly to the OBIS international data repository and portal, using GBIF IPT technology. The ALA data processing pipeline harvests, aggregates and enhances datasets from many sources with authoritative taxonomic and spatial reference data before passing the data on to GBIF. OBIS-AU and ALA are working together to ensure that the publication pathways for any datasets managed by both (with potential for duplication of records and incomplete metadata harvests) are rationalised and that a single collaborative workflow across both units is followed for publication to GBIF. Recently, the data management groups have established an agreement to cooperatively publish marine data and eDNA data. OBIS-AU have commenced publishing datasets directly to GBIF with ALA endorsement. We present the convergent evolution of OBIS and GBIF data publishing in Australia, adaptive data workflows to maintain data and metadata integrity, challenges encountered, how domain expertise ensures data quality and the benefits of sharing data skills and code, especially in publishing eDNA data types in DwC (using the DNA-derived data extension) and exploring the new CamTrap Data Package using Frictionless data. We also present the work that both data groups are doing toward adopting the GBIF new Unified Data model for publishing data. This Australian case study demonstrates the strengths of collaborative data publishing and offers a model that minimises replication of data in global aggregators through the development of regional integrated data publishing pipelines.

Knowledge Base on Species Life Traits: A Spanish/French Plinian Core implementation use case

The French “Traits” working group was created in 2021 to support the development of the national knowledge base on species life traits managed by the PatriNat department*1, to identify and implement a suitable standard for managing and sharing species life traits (including interactions) at the national, then international, level. Its core members are part of several PatriNat teams (Species Knowledge, Dissemination & Mediation, Coordination of Information Systems), as well as other French research units*2 working on the topic of traits and ontologies. The Plinian Core (Plinian Core Task Group 2021) was first discussed in 2004 and its development began in 2005–2006 when the first version was deployed as a collaboration between InBIO*3 (Costa Rica) and GBIF Spain*4. It reuses and extends the Darwin Core vocabulary (Wieczorek et al. 2012, Darwin Core Maintenance Interest Group 2014) to describe different aspects of biological species information, that is, all kinds of properties or traits related to taxa, including biological and non-biological species traits. The Plinian Core was discussed with Dr Pando (convener of the TDWG Plinian Core Task Group*5) during one of the Traits working group meetings, and was found to be relevant to the French species life traits database (currently in development). The Traits working group future works will be following the example of the Plinian Core-based EIDOS database*6 (Spanish Ministry for the Ecological Transition), which allows for detailed species pages with distinct information sections (e.g., interactions, taxonomy, legal status, conservation). This collaboration resulted in a Capacity Enhancement Support Programme project submission (GBIF 2023) between French and Spanish partners, allowing for the consolidation of both the infrastructure and the sharing process of species life traits for taxa found on all French territories, as well as European Union territories. Additionally, this is an opportunity to provide information to GBIF (Global Biodiversity Information Facility) through a new update of the TAXREF (Gargominy 2022) national checklist, one of the core constituents of the GBIF Backbone Taxonomy (GBIF 2022). Species life traits and interactions will be added thanks to the new Plinian Core extension implemented on the GBIF Integrated Publishing Toolkit (IPT),*7 and an Atlas of Living Australia’s architecture BIE (Biodiversity Information Explorer) module*8 developed by Costa Rica in the context of a Capacity Enhancement Support Programme (CESP) project carried out with SIBBR*9 (GBIF Brasil).

Publishing Australian Marine Data to OBIS: Twenty Years of Lessons Learnt

In 2003, the Australian Antarctic Data Centre published the first Australian dataset of seabirds from the Southern Ocean to OBIS (Ocean Biodiversity Information System) via DiGIR (Distributed Generic Information Retrieval). The dataset initially had 17 fields with an emphasis on counts of individuals. Standards evolved and with the development of the IPT (Integrated Publishing Toolkit) by GBIF (Global Biodiversity Information Facility) around 2008, large datasets could be published. OBIS subsequently adopted the IPT as the preferred publishing tool for providers to use. In 2016, the Darwin Core Event core with the OBIS Extended Measurements and Facts extension was released (De Pooter et al. 2017), meaning that richer and more comprehensive datasets could be published via the IPT. It is only recently that the biological aggregators (e.g., OBIS, GBIF) are looking at enhancing functionality to report this data. The Australian OBIS Node (OBIS-AU), hosted by CSIRO NCMI (the Commonwealth Science and Industrial Research Organisation National Collections and Marine Infrastructure Business Unit) now manages an Australian region marine biodiversity IPT with 30 million records from over 450 datasets. In the last 12 months, using the GBIF DNA Derived Data Extention, the OBIS-AU Node has published extensive eDNA datasets to OBIS with sequences and DNA related metadata. OBIS-AU has developed tools and procedures to ensure that data is of the best possible quality before it is published. Issues covered include preventing the duplication of data, preserving context, enhancing data once published with improvements in publication schemas, matching taxa, and identification of temporal or spatial errors.

How We Developed a Data Exchange Format: Lessons Learned from Camera Trap Data Package (Camtrap DP)

Camera trapping has revolutionized wildlife ecology and conservation by enabling automated data acquisition, leading to the accumulation of massive amounts of camera trap data worldwide (Steenweg et al. 2016, Kays et al. 2020, Delisle et al. 2021). Although management and processing of camera trap-derived big data are becoming increasingly solvable with the help of scalable infrastructures, harmonization and exchange of the data remain limited, hindering its full potential. We therefore developed a new data exchange format, Camera Trap Data Package (Camtrap DP), to facilitate the exchange, harmonization and archiving of camera trap data at local and global scales (Camtrap DP Development Team 2023). Camtrap DP was developed with two guiding principles. It should: allow easy and interoperable data exchange and be developed openly and collaboratively. allow easy and interoperable data exchange and be developed openly and collaboratively. Camtrap DP structures the data in a simple model consisting of three tables (Deployments, Media, and Observations), which supports a wide range of camera deployment designs, classification techniques and analytical use cases. To describe these tables and the accompanying metadata, we adopted the Frictionless Standards, a collection of open specifications developed by the Frictionless Data project (Fowler et al. 2018), which offer a standardized way to describe datasets, data files and tabular data. Doing so, we did not have to reinvent how to express generic properties such as licenses, contributors, file formats, field names, data types, required values, controlled values, and relationships between tables. We expanded upon those to describe the necessary properties related to camera trapping, relying on existing standards such as Darwin Core (Wieczorek et al. 2012), Audiovisual Core (Audiovisual Core Maintenance Group 2023) and Data Cite Metadata Schema (DataCite Metadata Working Group 2021) where possible. This approach is not only efficient, but also facilitates interoperability: since a Camtrap DP is in essence a Frictionless Data Package, existing software tools can be used to read and validate data. We developed Camtrap DP openly, collaboratively, and with version control from the start. It is licensed under the permissive MIT license, allowing anyone to use it. Suggestions for change were and continue to be discussed in a public issue tracker on GitHub. These are incorporated only after review and automated testing. Once a number of changes have been adopted, a new version of the standard is released using semantic versioning. This allows Camtrap DP to evolve over time, while making sure that software and datasets referring to older versions of the standard are still valid. Equally important to the success of a data exchange format is community-wide adoption, which requires trust and implementation by existing systems. From the start, we have involved researchers as well as maintainers of software tools and management platforms from the camera trapping community. Using an iterative approach, they tested and provided feedback on Camtrap DP to make sure it met their requirements. To aid their understanding of the format, we provided a website, an example dataset that is versioned with the format, and an R package to read, explore and visualize Camtrap DP datasets (Oldoni et al. 2023). Through open development and outreach, we also managed to get the support from trusted and well-recognized organizations such as Biodiversity Information Standards (TDWG) and the Global Biodiversity Information Facility (GBIF). As a result, Camtrap DP can now be used as a data publication format in the Integrated Publishing Toolkit (Robertson et al. 2014).

Establishing Plankton Imagery Dataflows Towards International Biodiversity Data Aggregators

Over the last decade, plankton research has experienced extensive developments in automatic image acquisition for identifying and quantifying plankton species. This information is useful for the reporting of plankton occurrences and ecological data. Imagery instruments can vary in the way they sample (benchtop or in situ imagers) and the particle’s size range they target (see Lombard et al. (2019) for an extensive comparison of instruments and specifications). However, due to the wide variety of instruments and their (automatic) output data and formats, it is challenging to integrate datasets that originate from different sources. For this reason, we developed recommendations for plankton imagery data management, which can promote the ability to make these datasets as FAIR (Findable, Accessible, Interoperable and Reusable principles), as possible. The workflow presented here could inspire other Biodiversity Information Standards TDWG communities working with (automated) imagery data (e.g., camera traps) such as the Audubon Core and Machine Observations Interest Group. The recommended data format follows the OBIS-ENV-DATA format (De Pooter et al. 2017), a Darwin Core-based approach to standardise biodiversity data (Wieczorek et al. 2012) used in EurOBIS, the European node of the Ocean Biodiversity Information System (OBIS) and EMODnet Biology, the European Marine Biodiversity Data Network. However, this format does not include sufficient information for imagery data, therefore we propose the use of additional Darwin Core terms. For example, by including the terms identifiedBy, identificationVerificationStatus and identificationReferences in the Occurrence table, more clarity is reported regarding the uncertainty of the classification made by an algorithm. Thus, data providers can publish manually validated datasets or datasets produced by fully automated plankton identification workflows; and users can choose to use validated or not validated data. See in Suppl. material 1 a practical example on how to report an imagery dataset following the best practices. Moreover, the OBIS-ENV-DATA format allows the ingestion of additional information thanks to the use of the Darwin Core (DwC) Extended Measurement Or Facts or eMoF extension in the DwC Event core. The eMoF stores biotic, abiotic and sampling measurements and facts that are related to the Event and Occurrence table. An important aspect of this extension is that it includes standardised terms and controlled vocabularies, such as the British Oceanographic Data Centre (BODC) vocabularies, to standardise parameters that are not covered by DwC. The advantages of these is to unambiguously report information and to include those measurements that cannot be reported in the Event and Occurrence table (e.g., reporting abundance or biomass of plankton), and that are crucial to investigate ecosystem functioning questions. As a consequence, biodiversity data aggregators can extend their scope beyond species occurrence data. Fig. 1 summarises a typical dataflow that goes from imagery data acquisition to publication in several steps: Images are cropped and classified with software. This can be done in EcoTaxa, a web application that allows users to taxonomically classify images of individual organisms. Data is formatted in OBIS-ENV-DATA format. This format can be exported from EcoTaxa through its API. Data is submitted to EurOBIS via the IPT (Integrated Publishing Toolkit). Data is quality controlled by the BioCheck tool. Data in EurOBIS can flow to EMODnet Biology, OBIS and GBIF (Global Biodiversity Information Facility). Images are cropped and classified with software. This can be done in EcoTaxa, a web application that allows users to taxonomically classify images of individual organisms. Data is formatted in OBIS-ENV-DATA format. This format can be exported from EcoTaxa through its API. Data is submitted to EurOBIS via the IPT (Integrated Publishing Toolkit). Data is quality controlled by the BioCheck tool. Data in EurOBIS can flow to EMODnet Biology, OBIS and GBIF (Global Biodiversity Information Facility). Plankton imagery instrument operators now have the possibility to format their data following the best practices and recommendations for plankton imagery data management (Martin-Cabrera et al. 2022). After a dataset is formatted following these guidelines, it can be submitted to the international biodiversity data aggregators, EurOBIS, EMODnet Biology and GBIF. Additionally a (semi) automated dataflow is presented where data providers can classify images in EcoTaxa and export the data in the required formats using an API before submission to EurOBIS. The next steps are to disseminate these best practices, encouraging plankton imagery data generators to implement these workflows to share their data easily, enriching these data portals and encouraging cross collaborations to create data products covering broader geographic scales and plankton species.

Developing and Maintaining a National Biodiversity Data Infrastructure – An example from Norway

Biodiversity data infrastructures are fundamental to halting the ongoing loss of species and habitats. Here we provide an overview of the national biodiversity data infrastructure developed and implemented in Norway by The Norwegian Biodiversity Information Centre (NBIC). Key elements and properties of this infrastructure are highlighted and directions for future development are outlined. The overarching objective for the infrastructure is to make data on habitats and species in Norway available for policy and decision makers, researchers and the general public. Here we will focus on data on species and their distribution. The infrastructure is built as a modular system but with a high level of integration between the components. NBIC has the main responsibility for developing and managing the infrastructure in collaboration with natural history museums, research organizations, private companies and non-governmental organizations (NGOs), including the Norwegian node of the Global Biodiversity Information Facility (GBIF). The infrastructure includes a citizen science portal that gives both amateurs and professionals the possibility to report species sightings. The data from this portal together with data from natural history museums and other data providers are then made available though the Species Map Service website. The infrastructure also includes the Norwegian taxonomic backbone database and a trait bank that is under development. The trait bank is planned to contain both ecological traits but also other information about species such as Red List status. The infrastructure builds on a few simple principles. The exchange of data is based on the Darwin Core Standard and its extensions, or other open data standards. Each data owner is responsible for quality control, secure storage and management of their own data. The data owners publish the data using the Integrated Publishing Toolkit (IPT) developed by GBIF. NBIC then harvests all observations with spatial coordinates that fall within Norway and adds information about, for example, status on the Norwegian Red List before the data are made available in the map. The core parts of the infrastructure only handle data that are open and adhere to the FAIR principles, i.e. that the data are Findable, Accessible, Interoperable, and available for Reuse. An important exception to this principle is observations of threatened species that are particularly vulnerable to human disturbance. Sensitive data are managed in a separate, secure system and with a restricted access portal for data viewing. Data in the Species Map Service are also available through a public API, which can be used to harvest and import the data or subsets of it into other systems and services. To give an example, the forestry industry integrates data on Red Listed species into their planning systems tools and has even an option to display a subset of species occurrences on their forestry machine computers. NBIC has in cooperation with NGOs established a system in which experts validate observations that are reported through the citizen science portal. As it is not possible to validate all observations, species of particular interest for environmental management and conservation are given priority. A future solution could be to develop a system that can identify observations with a low likelihood of being correct using, for example, statistical models that describe the likelihood of an occurrence in a given space and environment. If photos are available, image recognition based on machine learning can be used to spot species that are likely to be misidentified. It is also possible/conceivable to use a more heuristic approach based on an ontology for organisms and ecological traits (for example, "this is a fish, fish are always aquatic organisms, this observation was made on dry land and is thus unlikely to be true"). The development of improved tools and systems to improve data quality is a major task for future development of the infrastructure. In Norway, spatial planning processes and decisions are required to integrate information made available through the infrastructure. If new information about the occurrence of Red Listed species or other species of special interest for environmental management and conservation are discovered through the planning process, the developer is responsible for archiving these observations in a publicly available database. Key factors for the success of the biodiversity data infrastructure described here are: (i) the close cooperation between NBIC, data providers, NGO's and data users, (ii) the modularity of the infrastructure, (iii) the use of open and flexible standards for data exchange, and (iv) the integration of the infrastructure in the legal framework for spatial planning requiring that data from e.g., environmental impact assessments are made available through the infrastructure.

Leveraging the Symbiota Support Hub for Biodiversity Data Mobilization

Symbiota is an open source software for managing and mobilizing biodiversity data from physical and virtual collections. Over 700 natural history collections use Symbiota as their primary content management system, and over 600 additional collections use Symbiota portals to mobilize a copy or subsamples of their data for use by specific communities of expertise. For both "live-managed" and "snapshot" collections, Symbiota provides data import, export, and publishing tools to lower data mobilization barriers. For example, collections in Symbiota portals can publish their data directly to the Global Biodiversity Information Facility (GBIF) and Integrated Digitized Biocollections (iDigBio) using Darwin Core Archive protocols for data sharing, obviating the need to install or host local Integrated Publishing Toolkit (IPT) instances. Over 150 institutions currently use this workflow to publish datasets to GBIF. The strengths of Symbiota's approach to mobilization are part technical and part social. Once a collection has data in a portal, using the portal’s built-in data mobilization tools requires little technical expertise, and Symbiota portal managers are typically very accessible and helpful in assisting with the process. With sustained funding of the iDigBio Symbiota Support Hub, we are increasing the capacity of Symbiota to mobilize biodiversity through (1) improved documentation regarding the data mobilization process (https://symbiota.org/docs); (2) increased capacity for individualized attention to users through a larger service team and a robust help desk system; (3) the launch of "Portal Advancement Campaigns"—targeted efforts to promote data quality and mobilization (e.g., publishing to data aggregators); and (4) the development of an API infrastructure to enhance data interoperability and accessibility. We provide an overview of these new technologies and services for promoting and assisting with data mobilization, and we discuss future developments.

A Pre-ingestion Framework for Darwin Core Archives

The Atlas of Living Australia’s (ALA) Pre-ingestion Framework is our alternative to managing datasets via the Global Biodiversity Information Facility's (GBIF) Integrated Publishing Toolkit (IPT). The framework uses a system-agnostic Python codebase to create and update Darwin Core archives: building an archive from a core and extension csv files, merging two archives together, deleting records and identifying duplicates based on the identifiers. The framework dynamicly supports current Darwin Core and GBIF namespace terms. Previously, this functionality was handled internally by a Java-based biocache-store ingestion application. While flexible and easy to call, this black box approach to data management created challenges like removing problem records and tracking and verifying data sources. Last year, as the ALA merged our ingestion codebase with GBIF's pipelines and upgraded our data store infrastructure, we took the opportunity to manage our source data exclusively as full Darwin Core archives, rather than partial text files or spreadsheets. Consequently, the Python-based framework consolidates a lot of work previously managed using a range of methodologies and technologies including Talend, Java and unix based scripting. Alongside the Darwin Core archive manipulation tools, it has handlers for harvesting data from secure external web services, web hosts or file servers. The standardised approach to data loading paves the way for improved automation and workflow. The work has the potential to become an open source project to share with the Living Atlas and biodiversity informatics communities.

Taiwan Biodiversity Information Facility (TaiBIF)’s Role in Facilitating Open Biodiversity Data in Taiwan

Taiwan Biodiversity Information Facility (TaiBIF) was established in 2001 as one of the first Global Biodiversity Information Facility (GBIF) nodes. Funded by Taiwan's Ministry of Science and Technology and Forestry Bureau, TaiBIF’s missions are to: Develop infrastructures for biodiversity-related data, Promote the open data concept, Facilitate the publication and integration of biodiversity information, Support biodiversity-related research and policy-making, and Connect Taiwan and international biodiversity information. Develop infrastructures for biodiversity-related data, Promote the open data concept, Facilitate the publication and integration of biodiversity information, Support biodiversity-related research and policy-making, and Connect Taiwan and international biodiversity information. As part of the work to fulfill its missions, TaiBIF develops and maintains infrastructures such as a data portal, Taiwan Catalogue of Life (TaiCoL), and a tool for scientific name-checking - NomenMatch, for public and researchers to use as needed. Additionally, as the only GBIF data hosting centre currently available in Asia, TaiBIF offers all services mentioned in the GBIF’s data hosting criteria. Briefly, this includes a regularly maintained Integrated Publishing Toolkit (IPT) and help desk for supporting users in dataset and data paper publishing process. In the continuous effort to promote open biodiversity data, TaiBIF organises workshops annually for students, researchers, and any governmental or non-governmental organisations, providing hands-on training focusing on data standardising and the publishing process. As of June 2022, Taiwan has exposed more than 13 million occurrence records and 83 datasets through GBIF's data platform, contributed mostly by 19 data publishing units, making it the second-largest data contributor in Asia (after India). These data have been used in many peer-reviewed studies on various topics and for guiding conservation policies. TaiBIF has maintained close interactions and cooperation with GBIF, regularly assisting in the Biodiversity Information Fund for Asia (BIFA) projects, regional workshops and communications between members from Asian countries. TaiBIF’s current active tasks include allying with the Taiwan Biodiversity Information Alliance (TBIA), a national biodiversity network aimed at the integration of local biodiversity data from various governmental agencies and museums. Furthermore, a volunteer training system is being developed to support workloads at the TaiBIF office. Recognising the importance of data coverage, a data review study of open biodiversity data status in Taiwan is ongoing.

Plinian Core: The long and winding road

Plinian Core (PliC) is a set of vocabulary terms that can be used to describe different aspects of biological species information. Under "biological species information" all kinds of properties or traits related to taxa—biological and non-biological—are included. Thus, for instance, terms pertaining to descriptions, legal aspects, conservation, management, demographics, nomenclature, or related resources are incorporated. Plinian Core started as a collaborative project between Instituto Nacional de Biodiversidad (INBio, Costa Rica) and the Global Biodiversity Information Facility Spain (GBIF Spain) in 2005. In successive years, the National Commission for Knowledge and Use of Biodiversity (CONABIO, Mexico), the National Network of Open Biodiversity Data (SiB Colombia)/the Humboldt Institute (Colombia), the University of Granada (Spain) and the University of São Paulo (USP, Brazil) were also involved. In 2012. PliC aligned with the Biodiversity Information Standards (TDWG) and an umbrella Interest Group was created (Species Information IG, Pando 2017b) with developments carried out by its corresponding TDWG Task Group (Plinian Core Task Group 2020b). PliC design requirements include: ease of use, being self-contained, able to support data integration from multiple databases, and ability to handle different levels of granularity. Since its early stages, PliC was formalized as an XML Schema. Its terms can be grouped in its current version as follows: Metadata Base elements Record metadata Nomenclature and classification Taxonomic description Natural history Invasive species Habitat and distribution Demography and threats Uses, management and conservation Miscellaneous: associated party, measurement or fact, references and ancillary data Metadata Base elements Record metadata Nomenclature and classification Taxonomic description Natural history Invasive species Habitat and distribution Demography and threats Uses, management and conservation Miscellaneous: associated party, measurement or fact, references and ancillary data During its years of existence, PliC has been implemented in several real-life contexts (e.g., Spanish Ministry for the Ecological Transition and the Demographic Challenge 2020, CRBIO 2020, SIB Colombia 2020), and a set of extensions for using PliC in GBIF Integrated Publishing Toolkit (IPT) was developed (Plinian Core Task Group 2020a). PliC has been featured in some TDWG conferences, either to explore and explain how it relates to other TDWG standards (Pando 2018), how PliC handles specific information aspects (Pando 2017a), or how it may be used in biodiversity portals (Vargas et al. 2018, Vargas et al. 2019). Recent work has focused on how to represent terms and codify structure as an XML Schema (XSD) under the current TDWG Documentation Standard (SDS, Vocabulary Maintenance Specification Task Group 2017). Challenges identified in the process comprise: how to handle terms used in different contexts, how to preserve the semantic context of terms reflected in the XSD hierarchy in the SDS flat layout, and how to refer to terms and definitions defined in other data specifications. These are the areas to focus our contribution.

Application of Humboldt Extension to Real-world Cases

Access to high-quality ecological data is pivotal to assessing and modeling biodiversity and its change through space and time. Inventory data (i.e., recording multiple species at specific places and times) are particularly relevant to monitoring species distributions and abundance, but their reliability for use in downstream models depends on reporting the methodology implemented and associated sampling effort and completeness. This information about the inventory processes is often either not reported or described in an unstructured manner, greatly limiting potential re-use for larger-scale analyses. In order to support the reuse of inventories and to assure better standardization of newly collected data, we developed a framework to standardize inventory data reporting that is general enough for broad use. Guralnick et al. (2018) introduced the Humboldt Core as a proof of concept. In 2021, the TDWG Humboldt Core Task Group was established to review how to best integrate the terms proposed in the original publication with existing standards and implementation schemas. In the context of sharing data using the Darwin Core standard (DwC), different types of inventories can be represented as Events with different nesting levels. Therefore, it was deemed appropriate to develop an extension to DwC that allows capturing the details of the inventory process. The Task Group members revised all original terms, reformulated definitions, and discarded or added new terms where needed. We are developing a user guide and reaching out to the larger biodiversity community to test the Humboldt Extension with real-world case study datasets using a test instance of the GBIF Integrated Publishing Toolkit (IPT). In this presentation, we will review the development process, give an overview of how the Humboldt Extension can be used to report key information on the inventory process, and provide example cases. After testing with real world cases, our next step will be to seek ratification of Humboldt as a Darwin Core Event extension following the Vocabulary Maintenance Standard. We expect that this will help to overcome a key bottleneck in the sharing of critically important ecological data, enhancing data discoverability, interoperability and re-use while lowering reporting burden.

MaterialSample and its Properties

The Biodiversity Information Standards (TDWG) Material Sample Task Group*1 kicked off in the third quarter of 2021. The group’s initial focus was to 1) achieve a clear conceptual delineation between the terms MaterialSample, PreservedSpecimen, LivingSpecimen, and FossilSpecimen (the terms used in basisOfRecord in the current DwC-A provided to the Integrated Publishing Toolkit (IPT) for describing physical material) 2) define the conceptual relationship between these terms and the term Organism 3) consider the possible implications of the activities towards the diversification of the Global Biodiversity Information Facility (GBIF) data model*2 and what standards already exist that should inform our work. Based on this conceptual work, the group is now developing a concrete proposal for a clarification of a MaterialSample class with its own properties. Our presentation will provide a brief review of the task group's progress and our thoughts about what comes next.

APIs: A Common Interface for the Global Biodiversity Informatics Community

Web APIs (Application Programming Interfaces) facilitate the exchange of resources (data) between two functionally independent entities across a common programmatic interface. In more general terms, Web APIs can connect almost anything to the world wide web. Unlike traditional software, APIs are not compiled, installed, or run. Instead, data are read (or consumed in API speak) through a web-based transaction, where a client makes a request and a server responds. Web APIs can be loosely grouped into two categories within the scope of biodiversity informatics, based on purpose. First, Product APIs deliver data products to end-users. Examples include the Global Biodiversity Information Facility (GBIF) and iNaturalist APIs. Designed and built to solve specific problems, web-based Service APIs are the second type and the focus of this presentation (referred to as Service APIs). Their primary function is to provide on-demand support to existing programmatic processes. Examples of this type include Elasticsearch Suggester API and geolocation, a service that delivers geographic locations from spatial input (latitude and longitude coordinates) (Pejic et al. 2010). Many challenges lie ahead for biodiversity informatics and the sharing of global biodiversity data (e.g., Blair et al. 2020). Service-driven, standardized web-based Service APIs that adhere to best practices within the scope of biodiversity informatics can provide the transformational change needed to address many of these issues. This presentation will highlight several critical areas of interest in the biodiversity data community, describing how Service APIs can address each individually. The main topics include: standardized vocabularies, interoperability of heterogeneous data sources and data quality assessment and remediation. standardized vocabularies, interoperability of heterogeneous data sources and data quality assessment and remediation. Fundamentally, the value of any innovative technical solution can be measured by the extent of community adoption. In the context of Service APIs, adoption takes two primary forms: financial and temporal investment in the construction of clients that utilize Service APIs and willingness of the community to integrate Service APIs into their own systems and workflows. financial and temporal investment in the construction of clients that utilize Service APIs and willingness of the community to integrate Service APIs into their own systems and workflows. To achieve this, Service APIs must be simple, easy to use, pragmatic, and designed with all major stakeholder groups in mind, including users, providers, aggregators, and architects (Anderson et al. 2020Anderson et al. 2020; this study). Unfortunately, many innovative and promising technical solutions have fallen short not because of an inability to solve problems (Verner et al. 2008), rather, they were difficult to use, built in isolation, and/or designed without effective communication with stakeholders. Fortunately, projects such as Darwin Core (Wieczorek et al. 2012), the Integrated Publishing Toolkit (Robertson et al. 2014), and Megadetector (Microsoft 2021) provide the blueprint for successful community adoption of a technological solution within the biodiversity community. The final section of this presentation will examine the often overlooked non-technical aspects of this technical endeavor. Within this context, specifically how following these models can broaden community engagement and bridge the knowledge gap between the major stakeholders, resulting in the successful implementation of Service APIs.

Connecting West and Central African Herbaria Data: A new Living Atlases regional data platform

The label transcription and imaging of specimens in key African herbaria has been ongoing since the early 2000s. Many collections in Benin, Cameroon, Côte d’Ivoire, Gabon, Guinea Conakry, and Togo are now fully transcribed and partially digitized. More than 200 000 transcribed specimens are available with the following distribution: Benin: 45 000 Cameroon: 70 000 Côte d’Ivoire: 18 000 Gabon: 70 000 Guinea Conakry: 5 000 Togo: 15 000 Benin: 45 000 Cameroon: 70 000 Côte d’Ivoire: 18 000 Gabon: 70 000 Guinea Conakry: 5 000 Togo: 15 000 In April 2021, a BID project was started to deliver a regional data platform of West and Central African herbaria. Biodiversity Information for Development (BID) is a multi-year programme funded by the European Union and led by GBIF with the aim of enhancing capacity for effective mobilization and use of biodiversity data in research and policy in the 'ACP' nations of sub-Saharan Africa, the Caribbean and the Pacific. Our project's funding runs from April 2021 to April 2023. At this stage of the project, we are working on defining the information technology (IT) architecture (Fig. 1) and selecting the tools that we will be using to achieve our goals. In the talk, we will present our conclusions through architecture schemas and tools demonstrations. Each of the 6 countries will have its own PostgreSQL database, storing its data. They will also have access to the RIHA data management platform (Réseau Informatique des Herbiers d'Afrique / Digital Network of African Herbaria). This is a web application, developed in PHP, allowing full management of the data by herbarium administrators (Fig. 2). An Integrated Publishing Toolkit (IPT) will fetch these herbaria data from the databases, create the Darwin Core archives, and connect these data automatically to gbif.org on a periodic basis (Fig. 3). On the databases, we will use a PostgreSQL view to ease conversion from the RIHA data model to the Darwin Core model. On the IPT, we will create one dataset per country, linked to each PostgreSQL view. The SQL query will be configured to only fetch validated data, depending on the herbarium administrator's validation in the RIHA platform. The automatic and periodic data transmission to gbif.org is a feature available in the IPT, and recently improved by the GBIF France team, which contributes to the IPT development. Another part of the automatic data workflow will be to feed a Living Atlases portal for the West and Central African herbaria. This web application will allow public users to search, display and download herbaria data from West and Central Africa (Fig. 4). Internally, this Living Atlases application will reuse open source modules developed by the Atlas of Living Australia (ALA). The application is mainly written in Java, uses JQuery/Bootstrap for the interface and relies on SolR and Spark in the backend. It has been developed to be easily reusable, by only modifying configuration and doing web customization (HTML / CSS), hiding most of the backend technological complexity. The automatic data workflow will transfer datasets generated by the IPT, in Darwin Core Archive format, to the Living Atlases portal backend. A technical task orchestrator, yet to be selected, will implement this feature. Living Atlases subportals, limited to data of one participating country, could be easily set up, leveraging the existing backend resources (Fig. 5). One of the benefits of the Living Atlases portal is that we can easily deploy additional front end applications with limited data, configured by a filter (here, a filter on the data owner country). Only configuration and web customization (HTML / CSS) are required. All the backend modules, especially the ones storing data, are shared by the multiple front-ends, limiting the hardware consumption and data administration. The full automation of the workflow will allow this platform to run at a very low maintenance cost for IT administrators. Moreover, adding a new herbarium member from West and Central Africa will be quite easy thanks to the architecture of the Integrated Publishing Toolkit and Living Atlases tools (Fig. 6).

A Google Sheet Add-on for Biodiversity Data Standardization and Sharing

For those biologists and biodiversity data managers who are unfamiliar with information science data practices of data standardization, the use of complex software to assist in the creation of standardized datasets can be a barrier to sharing data. Since the ratification of the Darwin Core Standard (DwC) (Darwin Core Task Group 2009) by the Biodiversity Information Standards (TDWG) in 2009, many datasets have been published and shared through a variety of data portals. In the early stages of biodiversity data sharing, the protocol Distributed Generic Information Retrieval (DiGIR), progenitor of DwC, and later the protocols BioCASe and TDWG Access Protocol for Information Retrieval (TAPIR) (De Giovanni et al. 2010) were introduced for discovery, search and retrieval of distributed data, simplifying data exchange between information systems. Although these protocols are still in use, they are known to be inefficient for transferring large amounts of data (GBIF 2017). Because of that, in 2011 the Global Biodiversity Information Facility (GBIF) introduced the Darwin Core Archive (DwC-A), which allows more efficient data transfer, and has become the preferred format for publishing data in the GBIF network. DwC-A is a structured collection of text files, which makes use of the DwC terms to produce a single, self-contained dataset. Many tools for assisting data sharing using DwC-A have been introduced, such as the Integrated Publishing Toolkit (IPT) (Robertson et al. 2014), the Darwin Core Archive Assistant (GBIF 2010) and the Darwin Core Archive Validator. Despite promoting and facilitating data sharing, many users have difficulties using such tools, mainly because of the lack of training in information science in the biodiversity curriculum (Convention on Biological Diversiity 2012, Enke et al. 2012). However, most users are very familiar with spreadsheets to store and organize their data, but the adoption of the available solutions requires data transformation and training in information science and more specifically, biodiversity informatics. For an example of how spreadsheets can simplify data sharing see Stoev et al. (2016). In order to provide a more "familiar" approach to data sharing using DwC-A, we introduce a new tool as a Google Sheet Add-on. The Add-on, called Darwin Core Archive Assistant Add-on can be installed in the user's Google Account from the G Suite MarketPlace and used in conjunction with the Google Sheets application. The Add-on assists the mapping of spreadsheet columns/fields to DwC terms (Fig. 1), similar to IPT, but with the advantage that it does not require the user to export the spreadsheet and import it into another software. Additionally, the Add-on facilitates the creation of a star schema in accordance with DwC-A, by the definition of a "CORE_ID" (e.g. occurrenceID, eventID, taxonID) field between sheets of a document (Fig. 2). The Add-on also provides an Ecological Metadata Language (EML) (Jones et al. 2019) editor (Fig. 3) with minimal fields to be filled in (i.e., mandatory fields required by IPT), and helps users to generate and share DwC-Archives stored in the user's Google Drive, which can be downloaded as a DwC-A or automatically uploaded to another public storage resource like a user's Zenodo Account (Fig. 4). We expect that the Google Sheet Add-on introduced here, in conjunction with IPT, will promote biodiversity data sharing in a standardized format, as it requires minimal training and simplifies the process of data sharing from the user's perspective, mainly for those users not familiar with IPT, but that historically have worked with spreadsheets. Although the DwC-A generated by the add-on still needs to be published using IPT, it does provide a simpler interface (i.e., spreadsheet) for mapping data sets to DwC than IPT. Even though the IPT includes many more features than the Darwin Core Assistant Add-on, we expect that the Add-on can be a "starting point" for users unfamiliar with biodiversity informatics before they move on to more advanced data publishing tools. On the other hand, Zenodo integration allows users to share and cite their standardized data sets without publishing them via IPT, which can be useful for users without access to an IPT installation. Additionally, we are working on new features and future releases will include the automatic generation of Global Unique Identifiers for shared records, the possibility of adding additional data standards and DwC extensions, integration with GBIF REST API and with IPT REST API.

Linked Open Biodiversity Data (LOBD): A semantic application for integrating biodiversity information

Scientists frequently collect biological and environmental information over years and store it in database systems to answer their own research questions without exposing it in repositories that make it easy to find and retrieve. While in recent years the community working on biodiversity informatics has made significant strides by creating common shared vocabularies such as the Darwin Core (DwC, Wieczorek et al. 2012) and publishing mechanisms such as the Integrated Publishing Toolkit (IPT, Robertson et al. 2014), integration is largely limited to the aggregation of datasets and full interoperability has still not been achieved. In this context, The Semantic Web (SW) aims to represent information in a way that, in addition to the human-centered display purposes, it can be used autonomously by machines for integration and reuse across applications. From the biodiversity informatics point of view, interoperability and links among data sources would allow integration of information that is otherwise disconnected, enabling scientists to answer broader questions. These considerations provide strong motivations to formulate a web application considering the semantic interoperability that may provide answers to questions such as the following: (Q1) Is it possible to complement taxonomic, bibliographic and environmental information of a particular species without relying on specific Application Programming Interfaces (APIs)? (Q2) How to relate occurrences of species with environmental variables within a specific region? (Q3) What are the bibliographic references associated with a given species? (Q1) Is it possible to complement taxonomic, bibliographic and environmental information of a particular species without relying on specific Application Programming Interfaces (APIs)? (Q2) How to relate occurrences of species with environmental variables within a specific region? (Q3) What are the bibliographic references associated with a given species? With questions such as these in mind, we present the design of a proof-of-concept application: Linked Open Biodiversity Data (LOBD). LOBD uses Linked Data (LD) (Heath and Bizer 2011) to complement species occurrence information previously extracted from GBIF and converted to Resource Description Framework (RDF) (Zárate et al. 2020) with information about the taxa in question from different RDF datasets, such as Wikidata, NCBI Taxonomy, Springer Nature SciGraph and OpenCitation corpus. A simplified view of the architecture is shown in Fig. 1. To achieve semantic interoperability, we use the SPARQL query language, which allows us not to depend on specific APIs to retrieve information. The application consists of three modules: General information, where the Wikidata endpoint is used to retrieve additional information about the selected species, including links to other databases and information about the species extracted from National Center for Biotechnology Information (NCBI) Taxonomy. Bibliography, where all publications related to the species are retrieved and extracted from OpenCitation. Environment, where users can plot species on a map and add layers related to marine regions as well as environmental layers (e.g., temperature, salinity, etc). General information, where the Wikidata endpoint is used to retrieve additional information about the selected species, including links to other databases and information about the species extracted from National Center for Biotechnology Information (NCBI) Taxonomy. Bibliography, where all publications related to the species are retrieved and extracted from OpenCitation. Environment, where users can plot species on a map and add layers related to marine regions as well as environmental layers (e.g., temperature, salinity, etc). For the development of the application, we use the Shiny framework for R, access to SPARQL endpoints is done through the SPARQL package, marine regions are obtained from marineregion.org and the environmental layers are extracted from Bio-ORACLE. The data used for this article were collected by the Center for the Study of Marine Systems at the National Patagonian Sci-Tech Centre (CCT CENPAT-CONICET), and are published and available through the GBIF network. Linked Data is a powerful tool for scientists, as it allows generating new approaches to biodiversity informatics, which can help to address the data integration challenges. Users would benefit from complementing the current prevalent use of vocabularies that are not ontologically defined (like DwC) for sharing biodiversity data. Although this application is a proof of concept, it shows that with little effort, it is possible to achieve greater interoperability between datasets that were not initially represented as LD.

The e-Flora of South Africa – restructuring data to comply with Darwin Core standards for inclusion into the World Flora Online

The e-Flora of South Africa project was initiated in 2013 by the South African National Biodiversity Institute (SANBI) in support of the Global Strategy for Plant Conservation (GSPC, 2011-2020). South Africa's flora consists of ca. 21,000 taxa of which more than half are endemic. South Africa will contribute a national Flora towards Target 1 of the GSPC ("To create an online flora of all known plants by 2020"). South Africa's contribution is ca. 6% of the world’s flora of which ca. 3% are endemic and therefore unique. South Africa’s electronic Flora is comprised of previously published descriptions. South Africa’s e-Flora data forms part of the Botanical Dataset of Southern Africa (BODATSA) that is currently managed through the Botanical Research And Herbarium Management System (BRAHMS). To date, South Africa’s e-Flora data (http://ipt.sanbi.org.za/iptsanbi/resource?r=flora_descriptions) represents 19,539 indigenous taxa, 79,139 descriptions of distribution, morphological, habitat and diagnostic data, and 27,799 bibliographic records. The e-Flora data was recently published online using the Integrated Publishing Toolkit and henceforth harvested by the World Flora Online (WFO) into the portal. A series of challenges were encountered while manipulating descriptive data from BRAHMS to be ingested by the WFO portal; from taxonomic issues to data quality issues not excluding compliance to data standards. To contribute to the WFO portal, the taxa in BODATSA has to match with the taxa in the WFO taxonomic backbone. Once there is a match, a unique WFO taxon identifier is assigned to the taxa in BODATSA. This process presented various challenges because the WFO taxonomic backbone and the taxonomic classification system that is used by South Africa (South African National Plant Checklist) does not fully correlate. The schema used to store taxonomic data also does not agree between BRAHMS and WFO and had to be addressed. To enable consistency for future, a detailed guideline document was created providing all the steps and actions that should be taken when publishing an e-Flora, managed in BRAHMS, to the WFO portal. The presentation will focus on matching taxonomic classifications between BRAHMS and WFO; dealing with character encoding issues and manipulating data to meet Darwin Core standards.

How the Luxembourg Natural History Museum Has Established and Maintained a National Bio- and Geodiversity Data System

Over the past 20 years, the Luxembourg National Museum for Natural History (LMNH) has built a bio- and geodiversity information system to collate, manage and publish natural heritage observation and specimen data on a national and international level. To date the system counts over 2 million taxon occurrence and over 100,000 specimen records. The Museum has chosen, whenever available, public or open source software tools complying to international biodiversity data standards for recording, managing and publishing data to increase resilience, stay connected with community initiatives and mutualise development costs. A central component of the Museum’s national data hub is Recorder 6, a client-server database software for wildlife recording developed by the National Biodiversity Network in the UK. Today, the Recorder-Lux database contains a large portion of natural heritage information in Luxembourg and is synchronised daily into a publication database connected via the Integrated Publishing Toolkit (IPT) to the Global Biodiversity Information Facility (GBIF). Moreover, Recorder-Lux data is accessible via the national species mapping portal mdata.mnhn.lu which has been developed in-house and is aimed at scientists, professionals and decision makers. The Museum has also developed a set of data entry and upload functionalities on its website data.mnhn.lu using the open source software Indicia, a toolkit that provides a ready-made set of services and tools for online wildlife recording. In 2019, we implement the Atlas of Living Luxembourg (ALL) website all.mnhn.lu, based on the open source Atlas of Living Australia software. ALL is the most comprehensive data portal about natural heritage in Luxembourg, showing specimen data from the museum’s botany, zoology, paleontology, petrology and mineralogy collections as well as fungi, animal and plant observations collected from national and international organisations (via GBIF). Data providers vary from individual scientific collaborators to professional regional record centers or private consultancies working for public administrations. They use different tools offered by the museum to enter, manage and transfer their data to the system. Thus several regional record centers chose the client-server Recorder 6 software to manage and exchange their data, whereas individual scientific collaborators of the Museum enter or upload their data via the online data entry forms available on data.mnhn.lu. For large-scale, long-term, professional biodiversity monitoring and inventories at the national level, specific data entry forms and functionalities have been configured on the Indicia website. Finally, citizens can record species observations via the iNaturalist smartphone app. Due to the museum’s long history of conducting field inventories alongside collating and managing natural history collections, the data hub holds observation and collection data in one database. In 2003, the Museum has developed the Collection Management and Thesaurus extensions for the Recorder 6 software to catalogue, describe and manage specimens in the Museum collections. It allows handling of field-gathered data alongside specimen-specific data such as storage location, specimen type and conservation status. In recent years this has become an essential tool for the increasing effort directed at the digitisation of the diverse natural history collections at the Museum. Our small database team faces the challenge of integrating an ever increasing number of records from a variety of datasets, tools and initiatives. To keep the technical and administrative work as simple as possible we have implemented an open data policy and aim to increase the use of IPT to connect databases instead of physically importing all data into one central database. To improve data quality we focus on training experts to work with our Indicia verification tool.

Investment in the Long-Tail of Biodiversity Data: From local research to global knowledge

In business, the “long-tail economy” refers to a market strategy where the gravity center shifts from a few high-demand products to many, varied products focused on small niches. Commercialization of individually low-demand products can be profitable as long as their production cost is low and, all taken together, they aggregate into a big chunk of the market. Similarly, in the “business” of biodiversity data acquisition, we can find several mainstream products that produce zillions of bits of information every year and account for most of the budget allocated to increase our primary data-based knowledge about Earth’s biological diversity. These products play a crucial role in biodiversity research. However, along with these large global projects, there is a constellation of small-scale institutions that work locally, but whose contribution to our understanding of natural processes should not be dismissed. These information datasets can be collectively referred to as the “long-tail biodiversity data”. Here we present the case of the Museum of Sciences University of Navarra, which harbors the research activity of the Museum of Zoology (MZNA) and herbarium (PAMP) of the University of Navarra in Spain. For the last 40 years, its members have been involved in hundreds of research projects, from local to international level—but quantitatively, the vast majority of its biodiversity records come from Navarra, a smallish (10,000 sq. km) administrative region in the north of Spain. Despite its modest area, the available information about the region in the Museum database approaches one million records of thousands of species, including dozens of type series. Fifteen years ago, a series of national research grants enabled boosting digitization and public access to the database records through the Global Biodiversity Information Facility (GBIF). Although those grants were never renewed, the Museum continued its digitizing and standardizing program on vouchered collections, as well as sampling additional raw biodiversity data through long term ecological sites using the Museum’s resources, which annually provide thousands of new records at the local level. Currently, the Museum has already published through GBIF’s IPT (Integrated Publishing Toolkit) 30 datasets, containing more than half-million records of about 5700 taxa. Its records have contributed to more than 60 peer-reviewed publications over the last five years. Institutions that basically harvest biodiversity data at a local scale usually show a high degree of specialization, gathering researchers with strong (albeit often narrow) expertise in the taxonomy and ecology of nearby ecosystems. They are thus an extremely valuable tool when dealing with processes resulting in diversity changes that can be identified rather precisely, especially when their work can be traced back many decades. Making all this fine-scale information accessible and actionable requires, in most of the cases, a rather modest investment in staff training on data management (e.g., standards, database interoperability) or museum curation procedures, and on informatics infrastructure. As in the case of business, it is not a matter of choosing between producing blockbusters or independent cinema--but of leveraging available resources and maximizing output.